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Disclosure Statement 

The following organization and programmatic Root Cause Analysis has been prepared 
in accordance with the Mitsubishi Heavy Industries (MHI) corrective action program, 
which uses an after-the-fact hindsight-based analysis. The information identified in this 
evaluation was discovered and analyzed using all information and results available at 
the time it was written. These results and much of the information considered in this 
evaluation were not available to the organizations, management, or individuals during 
the period that relevant actions were taken and decisions were made. 

This evaluation does not attempt to make a determination whether any of the actions 
or decisions taken by management, internal organizations, or individual personnel at 
the time of the event was reasonable or prudent based on the information that was 
known or available at the time they took such actions or made such decisions. Any 
individual statements or conclusions included in the evaluation as to whether incorrect 
actions may have been taken or improvements are warranted are based upon all of 
the information considered, including information and results learned after-the-fact 
and evaluation in hindsight after the results of actions or decisions are known, and do 
not reflect any conclusion or determination as to the prudence or reasonableness of 
actions or decisions at the time they were made. 
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1.0 Executive Summary 

On January 31, 2012, after the replacement steam generators (RSGs) supplied by MHI 
had been operating for approximately 11 months, SONGS Unit 3 was brought into an 
unplanned shutdown due to primary to secondary leakage of approximately 82 
gallons/day in one RSG. The direct cause of the leakage was determined to be tube to 
tube wear in the free span section of the U-bend region of the RSG, leading to a leak 
from one of the tubes in that region. 

SONGS Unit 2 was in a refueling outage when the event occurred in Unit 3. During the 
normally scheduled outage inspections of the Unit 2 RSGs, tube wear was discovered 
in the vicinity of the retainer bars in the U-bend region of both RSGs. This wear was 
determined to have been caused by random vibration of the retainer bars. 

It was determined that all four RSGs experienced higher than expected tube wear. This 
wear is comprised of: (i) tube to tube wear in the tube free-span sections between the 
Anti-Vibration-Bars (AVBs) located in the U-bend region observed almost exclusively in 
Unit 3; (ii) tube to AVB wear, observed at discrete tube to AVB intersections, with no 
wear indications in the tube free-span sections (the tube to AVB wear indications are 
short in length, and are associated with small tube motions); (iii) tube to Tube Support 
Plate (TSP) wear; and (iv) retainer bar to tube wear. One RSG experienced minor tube 
wear from a foreign object, which has since been removed. 

MHI, working in conjunction with SCE personnel and other industry experts, 
determined the mechanistic causes of the tube wear. MHI formed a team composed of 
personnel from MHI and its U.S. subsidiary, plus outside consultants, to perform the 
Root Cause Analysis (RCA) of the tube wear identified in the SONGS Unit 2 and Unit 3 
RSGs. The two wear mechanisms that produced the deepest wear are evaluated in this 
report. They include: 

1. Tube to tube wear in the in-plane direction due to fluid-elastic instability (FEI) 

2. Retainer bar to tube wear due to turbulence induced vibration (also referred to 
as random vibration) and the low natural frequency of the retainer bar 

Additionally, because many tubes exhibit it, this report also addresses a third wear 
mechanism: 

3. Tube-to-AVB wear caused by turbulence induced vibration (also referred to as 
random vibration). 
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The RCA team used Cause-effect analysis, Barrier analysis and Change analysis to arrive 
at two Root Causes and three Contributing Causes. The Root Causes are: 

1. Insufficient programmatic requirement to assure effective AVB contact force to 
prevent in-plane fluid elastic instability and random vibration and subsequent 
wear under high localized thermal-hydraulic conditions (steam quality (void 
fraction), flow velocity and hydro-dynamic pressure). 

2. The design control process did not provide sufficient direction to assure that an 
evaluation of the need for an analysis of flow induced vibration of the retainer 
bar was performed and verified. 

The corrective actions to preclude repetition include: 

1. Revise Procedure 5BBB60-N01 "Procedure for Controlling of the Design 
Activities" to require that the need for effective tube to AVB contact force 
under high localized thermal-hydraulic conditions (steam quality (void fraction), 
flow velocity and hydro-dynamic pressure) be addressed in all MHI SG designs. 

l.a Further revise Procedure 5BBB60-N01 "Procedure for Controlling of the 
Design Activities" to require that sufficient contact force is assured under high 
localized thermal-hydraulic conditions (steam quality (void fraction) flow 
velocity and hydro-dynamic pressure), e.g., compare to the design parameters 
of previous successful MHI steam generator designs. 

2. Revise procedure 5BBB60-N01 "Procedure for Controlling of the Design 
Activities" to require that retainer bars and other steam generator parts subject 
to flow induced vibration be evaluated to determine the different analyses and 
the level of analysis that need to be performed to support the steam generator 
design. 



2.0 Background of the Incident 

2.1 Project Background 

In September 2004, MHI was awarded a contract to replace Southern California 
Edison's (SCE) original steam generators (OSGs) at Units 2 and 3 of the San Onofre 
Nuclear Generating Station (SONGS). The MHI-supplied replacement SGs (RSGs) had a 
number of differences from the OSGs provided by Combustion Engineering. One of the 
main differences was the substitution of Inconel 690 for Inconel 600 as the tube 
material. Inconel 690 is more resistant to corrosion than Inconel 600. However, Inconel 
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690 has a thermal conductivity approximately 10% less than that of Inconel 600. The 
requirement that the SG's thermal performance be maintained, in conjunction with 
maintaining a specified tube plugging margin, necessitated increasing the tube bundle 
heat transfer surface area from 105,000 ft 2 to 116,100 ft 2 (an 11% increase).The 
Certified Design Specification SO23-617-01, Rev. 3 stated that SCE intended to use the 
provisions of 10 C.F.R. §50.59 as the justification for the RSG design, which imposed 
physical and other constraints on the characteristics of the RSG design in order to 
assure compliance with that regulation. The RSGs were also required to fit within the 
same space occupied by the OSGs. 

The Certified Design Specification issued by SCE also required that MHI incorporate 
many design changes to minimize degradation and maximize reliability. The following 
are the design requirements specified for the U-bend supports: 

"3.10.3.5 ... The Supplier shall develop and submit for Edison's approval an 
Engineering and Fabrication Gap Control Methodology describing control of an 
effective "zero" tube-to-flat bar gap, gap uniformity and parallelism of the 
tube bundle in the out-of -plane direction prior to tube fabrication. The gap 
statistical size (mean value +3sigma ) shall not exceed 0.003", and shall be 
validated by empirical data. " 

The Unit 2 RSGs were delivered to SONGS in February 2009 and installed during a 
refueling outage between September 2009 and April 2010. The Unit 3 RSGs were 
delivered to SONGS in October 2010 and installed during a refueling outage between 
October 2010 and February 2011. 

On January 31, 2012, after the Unit 3 RSGs had been operating for approximately 11 
months, the unit was brought into an unplanned shutdown due to maximum primary 
to secondary leakage of approximately 82 gallons/day in one RSG. The direct cause of 
the leakage was determined to be tube to tube wear in the free span section of the 
U-bend region of the RSG, leading to a leak from one of the tubes in that region. 

Inspections of the Unit 2 RSGs(which was offline undergoing a refueling outage) 
revealed significant tube wear in the vicinity of the retainer bars in the U-bend region. 

In addition to these two forms of tube wear, all four RSGs were found to have 
experienced higher than expected tube to Anti-Vibration-Bar (AVB) and tube to Tube 
Support Plate (TSP) wear. One RSG had experienced minor tube wear due to a foreign 



object. 
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2.2 Technical Specification requirements potentially involved in the Problem 

Technical Specification (TS) 3.4.17 requires that SG tube integrity be maintained and 
that all SG tubes meeting the tube repair criteria be plugged in accordance with the 
Steam Generator Program. 

TS 5.5.2.11 requires a Steam Generator Program to be established and implemented to 
ensure that SG tube integrity is maintained. 

TS 5.5.2. 11. b specifies three performance criteria that must be met for SG tube 
integrity: 

1. "Structural integrity performance criterion: All in-service steam generator tubes 
shall retain structural integrity over the full range of normal operating conditions 
(including startup, operation in the power range, hot standby, and cool down and all 
anticipated transients included in the design specification) and Design Basis Accidents 
(DBAs). This includes retaining a safety factor of 3.0 against burst under normal steady 
state full power operation primary-to-secondary pressure differential and a safety 
factor of 1.4 against burst applied to the design basis accident primary-to-secondary 
pressure differentials. Apart from the above requirements, additional loading 
conditions associated with the design basis accidents, or combination of accidents in 
accordance with the design and licensing basis, shall also be evaluated to determine if 
the associated loads contribute significantly to burst or rupture. In the assessment of 
tube integrity, those loads that do significantly affect burst or rupture shall be 
determined and assessed in combination with the loads due to pressure with a safety 
factor of 1.2 on the combined primary loads and 1.0 on axial secondary loads." 

2. "Accident induced leakage performance criterion: The primary to secondary 
accident induced leakage rate for any DBA, other than a SG tube rupture, shall not 
exceed the leakage rate assumed in the accident analysis in terms of total leakage rate 
for all SGs and leakage rate for an individual SG. Leakage is not to exceed 0.5 gpm per 
SG and 1 gpm through both SGs." 

3. "The operational leakage performance criterion is specified in LCO 3.4.13, "RCS 
Operational Leakage." [This LCO is applicable in Modes 1-4 and states RCS operational 
leakage shall be limited to: (a) no pressure boundary leakage; (b) 1 gpm unidentified 
leakage; (c) 10 gpm identified leakage; and (d) 150 gallons per day (gpd) primary to 
secondary leakage through any one SG."] 
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3.0 Statement of Problem 

This Root Cause Analysis (RCA) was performed based on the following problem 
statement, which was adopted as part of the Root Cause Analysis Team Charter: 

(1) Requirement 

No Primary-to-Secondary Leakage due to Defects in any of the RSG Units for 
the duration of the Warranty Period, (per 17.2.3 of General T&C with EMS) 



(2) Deviation 

Unit 3 SG-B (SCE SG088) experienced tube leakage during operation and failure 
of eight tubes during in-situ pressure testing. (Both due to Defects) 



(3) Consequences (For MHI) 

r 



v. 



J 



4.0 Extent of Condition Evaluation 

To determine the extent of condition, other MHI SGs with similar design and 
construction were analyzed to see if the same tube wear conditions identified at the 
SONGS RSGs were present. 

The replacement steam generators for OPPD's Fort Calhoun Nuclear Generating 
Station are the only other steam generators designed by MHI operating in the United 
States. The OPPD RSGs replaced Combustion Engineering OSGs and are of a similar 
design and construction as the SONGS RSGs with certain differences, including: 



• Identical tube diameter (3/4") and wall thickness (0.043") 

• Identical tube pitch (1.0" equilateral triangle) 

• Identical pitch-to-diameter ratio (P/D = 1.33) 

• OPPD has greater average tube to AVB gap 

• OPPD RSGs are smaller than SONGS RSGs 

• Fewer AVBs than SONGS 

• Fewer tubes than SONGS 

• Smaller U-bend radius than SONGS 

• Lower maximum steam quality (void fraction) than SONGS 
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The Fort Calhoun RSGs have operated more than three fuel cycles with no evidence of 
U-bend tube degradation (no tube-to-AVB wear, no tube-to-tube wear, and no 
retainer bar-to-tube wear). Other steam generators designed by MHI (operating 
outside of the United States)are of a different design and have a variety of tube sizes, 
tube pitches and operating conditions. These steam generators have years of 
operation without significant tube wear. Therefore, it is concluded that the MHI SGs in 
operation today are not part of extent of condition. However, these other MHI SGs will 
be evaluated for susceptibility based on extent of cause. 

5.0 Analysis, Results, and Conclusions 

5.1 Evaluation Team Formation 

On March 23, 2012 MHI formed a team composed of personnel from MHI and its U.S. 
subsidiary, plus outside consultants, to perform the Root Cause Analysis of the tube 
wear identified in the SONGS Unit 2 and Unit 3 RSGs. The team was given the task of 
investigating the organizational and programmatic Root Causes of the tube wear. SCE 
also performed separate technical and Root Cause evaluations. 

The Root Cause Analysis commenced on March 26, 2012, and was conducted 
concurrently with the development of MHl's technical evaluation reports. 

5.2 Evaluation Methodology 

The evaluation team used the results of the technical investigations (identified below) 
as the basis for its analysis of the organizational and programmatic Root Causes for the 
tube to tube wear, retainer bar to tube wear, and tube to AVB wear seen in the RSGs. 
The extent of cause was evaluated based on organizational and programmatic causes. 

The team closely consulted with the MHI engineering team performing the technical 
evaluations, and with SCE representatives, in order to understand fully the technical 
causes of the tube wear. Additionally, the evaluation team gathered evidence through 
interviews, examination of procedures and plans and previous audits and surveillances, 
review of design and technical review meeting documents, and analysis of technical 
work products. 

To determine the organizational and programmatic Root and Contributing Causes of 
the three wear mechanisms evaluated in this report, the evaluation team used three 
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cause analysis tools: Cause-effect analysis, Barrier analysis, and Change analysis. The 
Root and Contributing Causes were determined primarily through the Cause-effect 
analysis. The results of the Barrier analysis and the Change analysis support the 
findings of the Cause-effect analysis. In addition to supporting the Cause-effect 
analysis, the Change analysis identified an additional Contributing Cause. 

In performing these analyses, the evaluation team closely looked at and took into 
account the technical evaluations prepared by MHI and SCE to understand fully the 
mechanistic causes of the tube to tube wear, the retainer bar to tube wear, and the 
tube to AVB wear, in order to better assess the underlying organizational and 
programmatic Root and Contributing Causes. The team then reviewed and evaluated, 
with the benefit of what is now known in hindsight, the design process for the RSGs to 
identify what could have been done differently that would have prevented the tube 
wear from occurring. Based on its reviews, the evaluation team identified the 
programmatic Root Causes of the RSG tube wear. 

5.3 Technical Investigation of the Incident 

MHI performed technical evaluations to identify the mechanistic causes of the tube 
wear, which identified fluid elastic instability as the mechanistic cause of the tube to 
tube wear, turbulence induced vibration (often referred to as "random vibration" 
because the excitation modes over time are unpredictable) as the mechanistic cause of 
the tube to AVB wear, and turbulence induced vibration of the retainer bar as the 
mechanistic cause of the retainer bar to tube wear. These evaluations are reflected in 
the MHI reports Tube Wear of Unit-3 RSG Technical Evaluation Report, L5-04GA564 
Rev.9; Retainer Bar Tube Wear Report, L5-04GA561 Rev.4; Validity of Use of the FIT-HI 
Results During Design, L5-04GA591 Rev. 3;and Supplemental Technical Evaluation 
Report, L5-04GA588 draft. SCE also performed Root Cause evaluations.SCE reports 
Root Cause Evaluation NN201843216 Steam Generator Tube Wear San Onofre Nuclear 
Generating Station, Unit 2dated April 2, 2012, and Roof Cause Evaluation: Unit 3 
Generator Tube Leak and Tube-to-Tube Wear Condition Report: 201836127, 
Rev.Ocontain the SCE Root Cause evaluations. 

The MHI and SCE mechanistic cause analysis reports used Fault Tree Analysis and 
Kepnor-Tregeo (respectively) as the primary analysis tools. Each of these analyses 
considered a broad range of potential causes. The following causes were evaluated in 
detail: 
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Manufacturing/fabrication 


Shipping 


Primary side flow induced vibration 


Divider plate weld failure and repair 


Additional rotations following divider 
plate repair 


TSP distortion 


Tube bundle distortion during operation 
(flowering) 


T/H conditions/modeling 



Each of these causes is evaluated in the MHI and SCE technical evaluation reports. 

These technical evaluations identified five different wear categories for the tubewear 
observed in the SONGS RSGs. Two of these wear categories are responsible for the 
most significant instances of tube degradation(in terms of the depth of wear and 
potential for failing to meet the technical specification requirements) and are being 
evaluated in this report to determine their organizational and programmatic causes. 
The two significant wear categories that are evaluated in this RCA are: 

1. Tube to Tube Wear due to in-plane FEI : Tube to tube wear was found in the 
U-bend region, located between AVBs, in the free span. Many of the tubes 
exhibiting tube to tube wear also exhibited wear at the AVBs and TSPs, in 
particular at the top tube support plate. For tubes with wear at the top tube 
support plate, it is considered that the entire tube, including its straight region, 
is vibrating. Tube to tube wear occurs when there is tube in-plane motion 
(vibration) with a displacement (amplitude) greater than the distance between 
the tubes in the adjacent rows, resulting in tube-to-tube contact. 1 

2. Retainer Bar to Tube Wear due to Flow Induced Vibration : Tube wear occurred 
on tubes at the periphery of the U-bend, adjacent to the retainer bars. These 
tubes have no wear indications at any other location along their length, which 



1 Some of the tubes with tube-tube wear did not experience large amplitude vibration 
but were impacted by tubes that did experience large amplitude vibration. Also the 
two tubes in Unit 2 with tube-to-tube wear had different wear characteristics than the 
Unit 3 tube-to-tube wear. 
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indicates that they are stationary, and that the wear is caused by the 
movement (vibration) of the retainer bars. 

Additionally, because many tubes have smaller-depth wear indications at the AVB 
intersections, this report also addresses another wear category: 

3. Tube to AVB Wear (for tubes without free span wear) due to random vibration : 
Tube wear occurred at discrete tube-to-AVB intersections, with no wear 
indications in the tube free-span sections. These wear indications are short in 
length and are associated with small tube motions. 

The other two categories of wear identified were: (i) wear at the TSPs (small bend 
radius tubes and tubes at the tube bundle periphery), and (ii) wear due to a foreign 
object. These two categories are not considered in this report because the degree of 
wear due to them is relatively small. 

The conclusions of the MHI and SCE technical evaluations have been accepted as the 
basis of this analysis. To the extent these evaluations are revised or amended to reflect 
additional information or new understandings, this evaluation may be affected. 

5.4 Description of Main Wear Mechanisms 
Fluid Elastic Instability 

In a tube array, a momentary displacement of one tube from its equilibrium position 
will alter the flow field and change the forces to which the neighboring tubes are 
subjected, causing them to change their positions in a vibratory manner. When the 
energy extracted from the flow by the tubes exceeds the energy dissipated by damping 
it produces fluid elastic vibration. 

Fluid Elastic Instability (FEI) is a term used to describe a range of tube vibrations that 
starts at a point on a curve of vibration amplitude versus flow velocity. As depicted in 
Figure 1, one axis (Y) of that curve is vibration amplitude and the other (X) is flow 
velocity. The graph shows that as flow velocity increases vibration amplitude increases 
at a small linear rate until it reaches a point where the slope of the curve increases 
abruptly. The point in the curve where the slope changes is termed "critical velocity". 
The critical velocity is a function of several variables. These include tube natural 
frequency, which is dependent on the tube geometry and support conditions, damping, 
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which is a function of the steam-to-water ratio, flow velocity, which is dependent of 
the tube spacing. 




Fluid Elastic 
Instability 



Random 
Vibration 



"Critical Velocity" 



Flow 
Velocity 



Figure 1 



As discussed below and in the technical reports referenced above (See Supplemental 
Technical Evaluation Report), MHI has determined that, due to ineffective support for 
the tubes in the in-plane direction resulting from the very small and uniform tube-to 
AVB gaps, some of the tubes exceeded the fluid elastic critical velocity resulting in 
in-plane FEI, which in turn produced the large amplitude tube-to-tube wear. This 
mechanism is influenced by the local thermal hydraulic conditions around the tube. 
Regions of high void fraction have lower tube damping, which reduces the fluid elastic 
critical velocity threshold. High void fraction regions also have higher cross flow 
velocities. Therefore, tubes with low or no contact force in the region of highest void 
fraction are most susceptible to this mechanism. 

Random Vibration 

Random vibration is the vibration mechanism caused by flow turbulence that changes 

proportionately to changes in the fluid flow forces(dynamic pressure) and is present at 

all flow velocities. Turbulent flow forces are random in nature, so this form of vibration 

is referred to as random vibration. As discussed below and in the technical reports 

referenced above, MHI has determined that the tube wear at the AVB intersections 
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with no wear indications in the tube free span sections is due to turbulence induced 
vibration caused by insufficient contact force between the tube and the AVBs due to 
very small, uniform tube-to-AVB gaps. Since dynamic pressure and damping is 
proportional to the void fraction, tubes in the region of highest void fraction are most 
susceptible to this mechanism. 

Tube to Tube Wear 

Tube-to-tube wear was caused by large displacements of tubes in the in-plane 
direction. Tubes are known to have moved in-plane because of the locations and 
magnitudes of their wear scars. The wear scars indicate that the tubes were generally 
vibrating in their first fundamental in-plane mode, which implies that none of the 
twelve (12) AVB supports were restraining the tube motion. Yet, it also indicates that 
the tube-to-AVB gaps are very small and uniform, because none of the tubes exhibited 
out-of-plane FEI, which is the tube's preferential fluid elastic vibration mode. 2 It can 
therefore be concluded that the tube-to-AVB contact forces were negligible and the 
tube-to-AVB gaps (on both sides of each tube at each of the 12 AVB intersections) 
were very small. Both of these conclusions are consistent with the original design 
intent discussed below. 

In-plane FEI is a phenomenon that had not been experienced in nuclear U-tube steam 
generators prior to its being identified in the SONGS RSGs. The practice in the nuclear 
industry at the time the SONGS RSGs were designed was to provide measures to 
preclude out-of-plane FEI in the U-bend region, which was based on the understanding 
set forth above. Reflecting this industry practice, the Japan Society of Mechanical 
Engineers' "Guideline for Fluid-elastic Vibration Evaluation of U-bend Tubes in Steam 
Generators" states that in-plane FEI does not need to be considered if out-of-plane FEI 
is controlled. The design of the SONGS RSGs is consistent with the contemporary 
industry practice and guidance. The RSGs were designed to provide effective tube 
support (by means of AVBs) to avoid out-of-plane FEI. MHI sought to maximize the 



2 ln U-bend SGs, because the tubes are curved, for the same support conditions the 
critical velocity for out-of-plane FEI will be lower than that for in-plane FEI because the 
natural frequency of tubes in the in-plane direction is higher, due to the tubes greater 
stiffness in-plane, than the natural frequency of the tubes in the out-of-plane 
direction. 
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adequacy of the supports against out-of-plane FEI by increasing the number of AVBs to 
a number, 12, that exceeds that in other U-tube SGs designed by MHI or by other 
major U-tube SG manufacturers. 

Minimizing tube vibration wear in the U-bend region was given high priority in the 
SONGS RSG Design Specification, the RSG design program, and in the manufacturing 
processes. Early in the project, SCE and MHI formed an AVB Design Team with the goal 
of minimizing U-bend tube vibration and wear. The AVB Design Team conducted 
numerous technical and design review meetings. The agreed-upon tube bundle 
U-bend support design and fabrication were as follows: 

• Six (6) V-shaped AVBs (three sets of two) were to be provided between each 
tube column (12 AVB intersections total around the U-bend). 

• Tube and AVB dimensional control, including increasing the AVB thickness was 
to achieve an effective "zero" tube-to-AVB gap under operating (hot) 
conditions with gap uniformity and parallelism being maintained throughout 
the tube bundle. Effective "zero" gap was desirable as an industry practice in 
order to maximize the effectiveness of the supports. The tube and AVB 
tolerances were to be tighter than that of any prior MHI SG. 

• Excessive preload contact force was to be avoided in order to minimize 
ding/dent indications, and to maintain mechanical damping and thus minimize 
tube vibration. 

MHI investigated field experience with U-bend tube degradation using INPO, NRC and 
NPE data bases, and concluded that the SONGS RSGs were designed to minimize the 
potential for tube wear by providing extra support points with shorter spans in the 
U-bend region along with effective zero tube-to-AVB gaps. 

In the fabrication process, MHI manufacturing focused on achieving very small, 
uniform tube-to-AVB gaps during assembly. 

The AVB Design Team included consultants with knowledge and experience in the 
design and construction large U-bend SGs. One consultant had experience with the 
design of a plant whose SGs were similar to the proposed RSGs (the "comparison" or 
"reference" plant). Together, the AVB Design Team concluded that the SONGS RSGs 
had more tube vibration margin than the comparison plant, which had experienced 
only a small number of tube wear occurrences. This conclusion was due to the 
following considerations^) SONGS RSG tubes are larger, have thicker walls, and are 
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stiffer than those of the comparison plant; (ii) the SONGS distances between AVB tube 
supports are shorter than those at the comparison plant; (iii) SONGS has 12 AVB tube 
supports where the comparison plant only has 10; (iv) SONGS's tube-to-AVB gap 
requirement was more stringent than that of the comparison plant. 

The Certified Design Specification SO23-617-01, Rev. 3, issued by SCE required an 
effective zero gap and gap uniformity and parallelism of the tube bundle in the 
out-of-plane direction. Establishing the goal to reduce tube-AVB gaps to an effective 
zero gap was in accordance with well accepted industry practice and understanding 
that minimizing gaps was highly desirable in preventing tube vibration wear. MHI had 
sought to minimize tube-AVB gaps in its previous SG designs. However, MHI took 
additional steps to minimize the tube-AVB gaps for the SONGs RSGs and to provide for 
gap uniformity throughout the U-bend region of the tube bundle. 

These steps included increasing the nominal thickness of the AVB compared to 
previous MHI SGs and reducing the manufacturing tolerance of AVB thickness and 
twist in order to achieve effective zero gaps and provide gap uniformity. Steps were 
taken as well to minimize tube ovality and to minimize variations from the design value. 
Also, numerous additional steps were taken in fabricating the tube bundle to assure 
gap uniformity throughout the U-bend region. Additionally, in the fabrication of the 
Unit 2 RSGs MHI identified other enhancements that were implemented in the 
fabrication of the Unit 3 RSGs. These included, for example, taking steps to minimize 
AVB twist by applying a larger(from£ jtons to£ jtons) pressing force in the Unit 
3 fabrication and thus providing for more uniform AVBs in the Unit 3 RSGs. 

The adequacy of the design against out-of-plane FEI was confirmed through test data 
and analyses that conservatively assumed that one of the AVBs provided in the design 
was inactive (that is, ineffective against out-of-plane FEIJ.Analyses using this criterion 
showed that an adequate margin against out-of-plane FEI exists in the SONGS RSGs. An 
additional AVB had been added to the design to provide further margin against 
out-of-plane FEI. 

The MHI technical evaluations performed after the January 2012 incident determined 
that, despite the robustness of the MHI design, in-plane FEI had occurred. This 
occurrence was due to a combination of a lack of effective contact forces between the 
tube and AVB in the in-plane direction and localized thermal-hydraulic (T/H) conditions 
(high steam quality (void fraction) and high fluid velocity).The evaluations found that 
the average contact force in the Unit 3 RSGs was smaller than the average contact 
force in the Unit 2 RSGs. Therefore, the contact forces of the Unit 3 RSGs were more 
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likely to be ineffective in preventing in-plane motion of tubes so that the Unit 3 RSGs 
were more susceptible to in-plane tube vibration than those in Unit 2. The difference 
in the contact forces between the Unit 2 and Unit3 RSGs is caused by the reduction in 
dimensional variations during the manufacture of the Unit 3 RSGs, mainly due to 
improvement of the control over tube and AVB dimensions in the manufacture of the 
Unit 3 RSGs. The reduced contact forces resulted in far more tubes in the Unit 3 RSGs 
experiencing tube-to-tube wear than those in the Unit 2 RSGs. For those tubes, given 
these support conditions, the vibratory energy in high localized thermal-hydraulic (T/H) 
environment produced in-plane FEI that led to large amplitude displacement of the 
tubes in the in-plane direction, which caused wear from contact between adjacent 
tubes. 

Tube Wear at AVBs 

Tube-to-AVB wear is a function of the amplitude of the random tube vibration and the 
tube-to-AVB gap. Where there is a gap between the AVB and the tube and the 
vibration amplitude is less than the gap, there will be minimal or no wear. If the AVB is 
in contact with the tube but there is insufficient contact force to lock the two together, 
there will be relative motion between the two and wear will occur. In the case where 
there is sufficient contact force to lock the two together, there will be minimal or no 
relative motion and only minimal wear will occur. In the SONGS RSGs, the zero gap 
design philosophy resulted in the AVBs being in contact with the tubes or very close to 
the tubes, but there was insufficient contact force to lock the two together, thus 
allowing tube wear at the AVBs. 

The degree of wear is also affected by the amount of damping provided by the water 
film between the tubes and AVBs. In the SONGS RSGs, damping was reduced in areas 
of high steam quality (void fractionjbecause there is less two-phase damping and little 
or no water film in the gaps between the tubes, resulting in more pronounced wear. 

Tube Wear at Retainer Bars 

The tubes exhibiting retainer bar wear have no indications of tube-to-tube or 
tube-to-AVB wear, which indicates that the wear is caused solely by retainer bar 
vibration. The SONGS RSGs have two types of retainer bars: £ 

diameter and [ jdiameter. Tube wear was only found on tubes adjacent 

to the smaller diameter retainer bars. The retainer bars with the smaller diameter have 
a relatively long span as compared with those for other SGs fabricated by MHI, which 
means that the natural frequency of these retainer bars is lower, making them more 
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likely to vibrate. This type of wear is caused by random flow-induced vibration of the 
retainer bars caused by the secondary fluid exiting the tube bundle. 



5.5 Discussion of Tube to Tube Wear 
Tube Contact Force 

During the fabrication of the AVBs and the tubing and assembly of the tube bundle, 
MHI's manufacturing practices achieved dimensional control that resulted in smaller 
tube-to-AVB gaps and smaller tube-to-AVB contact forces. It was not recognized at the 
time that a certain amount of tube-to-AVB contact force was required to prevent 
in-plane FEI under high steam quality (void fraction) conditions, because the contact 
force serves to increase the in-plane natural frequency of the tube. 

The technical investigations after the tube leak incident determined that the amount 
of contact force necessary to prevent in-plane FEI depends on the localized 
thermal-hydraulic conditions (steam quality (void fraction), flow velocity and 
hydro-dynamic pressure).As the steam quality (void fraction) increases, the amount of 
contact force necessary to prevent vibration increases. This increase in required 
contact force occurs because as the steam quality (void fraction) becomes higher, the 
damping provided by the liquid phase in the form of a liquid film decreases. 

The reduced in-plane contact force due to the SONGS "effective zero gap" design and 
the avoidance of "excessive preload" resulted in lowering the tubes' natural frequency 
in the in-plane direction. The combination of the localized high steam quality (void 
fraction) and reduced tube to AVB contact force resulted in exceeding the in-plane 
critical velocity, which created a condition that led to tube to tube contact. 

The dominant role played by the low contact force is reflected by the differences in the 

tube-to-tube wear that was observed in the Unit 2 and the Unit 3 RSGs. Each of the 

Unit 3 RSGs had approximately 160 tubes that experienced tube-to-tube wear whereas 

only one of the Unit 2 RSGs experienced tube-to-tube wear in just two tubes, even 

though the Unit 2 RSGs have operated twice as long as the Unit 3 RSGs. MHI did a 

comprehensive statistical evaluation of the contact forces between the tubes and the 

AVBs of the two units and concluded, based on the manufacturing data , that the 

contact force between the tubes and the AVBs in the Unit 2 RSGs is approximately 

double the contact force in the Unit 3 RSGs. Thus, the lower contact forces in Unit 3 

are consistent with the conditions determined necessary to permit in-plane FEI to 

occur and with the fact that tube-to-tube wear occurred almost exclusively in Unit 3. 
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Thermal-hydraulic Conditions 

Many analyses are performed during the steam generator design process. One of 
these is MHI's FIT-III tube bundle flow analysis, which calculates tube bundle thermal / 
hydraulic parameters, including U-bend flow velocity and steam quality (void 
fraction).An after-the-fact comparison between the T/H parameters that FIT-III 
predicted and those predicted by ATHOS, another T/H code, determined that FIT-lll's 
calculated values are lower than those obtained using ATHOS. Part of the difference 
was because the pressure loss coefficients for the tube bundle and the two-phase 
mixture density utilized in the two codes were different. 

Also, during the computation of the flow velocity, MHI used an inappropriate 
definition of the gap between tubes, with the result that the flow velocities were 
underestimated. 

These differences between MHI's use of the FIT-III model and the ATHOS model 
resulted in a higher margin to out-of-plane FEI than the margin that would have been 
determined using the appropriate the definition of the gap and an ATHOS-calculated 
steam quality (void fraction). The margin calculated using ATHOS, nonetheless, would 
still have resulted in adequate margin against out-of-plane FEI. Using the ATHOS 
outputs, with all AVBs assumed active, the stability ratio was less than 1.0 for 
out-of-plane FEI, even for those case studies assuming reduced damping that could 
occur under high void fraction conditions. 3 Thus, the use of ATHOS as opposed to 
FIT-III would not have identified an inadequate design margin against FEI. 

Moreover, because industry practice was focused on out-of-plane FEI, use of ATHOS 
would not have identified the potential for in-plane vibration. Both the academic 
literature and subsequently conducted tests show that the thermal-hydraulic 
environment under which in-plane FEI arises is different from those that result in 
out-of-plane FEI. (See Supplemental Technical Evaluation Report). If the steam quality 
(void fraction) predicted by FIT-III had been the same as the ATHOS calculated value, 



3 The maximum stability ratio based on ATHOS outputs for all supports are active 
is J, which is less than 0.75, which is the conservative industry practice for judging 
acceptability of stability ratios (which in turn is less than the ASME Section III Appendix 
N-1330 recommended stability ratio criterion of 1.0). Assuming reduced damping, the 
maximum stability ratio calculated using ATHOS is | ]. 
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and if the appropriate tube to tube gap value had been utilized to compute the flow 
velocity, MHI would have identified a decreased margin against out-of-plane FEI. In 
that case, MHI might have incorporated an additional AVB to increase the design 
margin against out-of-plane FEI, but would not have taken measures to protect against 
in-plane FEI, for it was assumed (as was the practice and guidance in the industry) that 
the controlling effect of a well-designed AVB system was adequate to preclude it. 

Thus, not using ATHOS, which predicts higher void fractions than FIT-III at the time of 
design represented, at most, a missed opportunity to take further design steps, not 
directed at in-plane FEI, that might have resulted in a different design that might have 
avoided in-plane FEI. However, the AVB Design Team recognized that the design for 
the SONGS RSGs resulted in higher steam quality (void fraction) than previous designs 
and had considered making changes to the design to reduce the void fraction (e.g., 
using a larger downcomer, using larger flow slot design for the tube support plates, 
and even removing a TSP). But each of the considered changes had unacceptable 
consequences and the AVB Design Team agreed not to implement them. Among the 
difficulties associated with the potential changes was the possibility that making them 
could impede the ability to justify the RSG design under the provisions of 10 C.F.R. 
§50.59. Thus, one cannot say that use of a different code than FIT-III would have 
prevented the occurrence of the in-plane FEI observed in the SONGs RSGs or that any 
feasible design changes arising from the use of a different code would have reduced 
the void fraction sufficiently to avoid tube-to-tube wear. 

For the same reason, an analysis of the cumulative effects of the design changes 
including the departures from the OSG's design and MHI's previously successful 
designs would not have resulted in a design change that directly addressed in-plane 
FEI. 

Summary 

Thus, the organizational and programmatic Root Cause for the in-plane FEI as set forth 
in this RCA is the insufficient programmatic requirement to assure effective AVB 
contact force to control in-plane FEI under high localized thermal-hydraulic conditions 
(steam quality (void fraction), flow velocity and hydrodynamic pressure). The 
underlying reason for this insufficiency is that the MHI SONGS RSG design did not 
consider the phenomenon of in-plane FEI because contemporary knowledge and 
industry U-tubeSG operation experience did not indicate a need to consider in-plane 
FEI. 
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5.6 Discussion of Tube to AVB Wear 

Tube-to-AVB wear in the SONGS RSG occurs at the tube-to-AVB intersections and is 
produced by turbulence induced (random) vibration. This population only includes 
tubes with wear at the tube-to-AVB intersections with no wear indications in the tube 
free-span sections. 

Tube wear at the AVB intersections (in the absence of tube-to-tube free span wear) 
occurs when the tube movement causes it to impact or slide along the supporting 
AVBs. The most common cause of this condition is out-of-plane FEI. In the SONGS RSG 
design, the large number of AVB supports and the superior gap control prevent 
out-of-plane FEI. However, because of the low contact forces between tubes and AVBs, 
the very small and uniform tube-to-AVB gaps, and the localized T/H conditions (high 
steam quality (void fraction) and high flow velocity), turbulent flow conditions are 
sufficient to produce tube wear at the AVB intersections. Again the effect of the 
different contact forces between Unit 3 and Unit 2 can be seen in the observed 
tube-to-AVB wear populations of the two units. Unit 2 had about two-thirds as many 
tube-to-AVB indications than Unit-3 and Unit 2 operated longer than Unit 3, indicated 
that the wear rate is greater at Unit 3. This is attributable to the lower contact forces. 
(See Supplemental Technical Evaluation Report). 

As was the case with tube-to-tube wear, it was not recognized at the time of the RSG 
design that a certain amount tube to AVB contact force is required to prevent random 
vibration under high localized thermal-hydraulic conditions (steam quality (void 
fraction), flow velocity and hydro-dynamic pressure).The combination of the reduced 
tube to AVB contact force and the localized T/H conditions (high steam quality (void 
fraction) and high flow velocity) resulted in tube to AVB wear. 

5.7 Discussion of Retainer Bar to Tube Wear 

The design function of the retainer bar is to support the AVB assembly during 
manufacturing and prevent excessive AVB assembly movement during operational 
transients. The retainer bar must be strong enough to support the AVB assembly and 
fit within the physical constraints of the U-bend. 

The tubesheet drilling pattern is one of the first design decisions made for a new steam 
generator and it is at that time that each tube location along the periphery of the tube 
bundle is established. The tube bundle design thus determines the retainer bar's 
length and thickness. At SONGS, in order to accommodate the increased number of 
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tubes, the retainer bars are relatively long and thin as compared to the retainer bars in 
other SGs designed by MHI, resulting in their having low natural frequencies. 

The engineer responsible for the retainer bar design did not recognize the need to 
analyze the retainer bar for flow induced vibration because no such analysis had been 
performed on previous MHI SG designs. The design control procedure for this design 
activity did not identify this issue, nor was it recognized during the design review 
process. 

During operation, the secondary flow velocity and steam quality (void fraction) created 
turbulent flow conditions capable of causing high amplitude vibration if the retainer 
bar natural frequency was low enough, which turned out to be the case. The high 
amplitude vibration resulted in the retainer bar contacting some tubes and causing 
tube wear. 

5.8 Root Causes 

As used in this evaluation, "Root Causes" are defined as the basic reasons (e.g., 
hardware, process, or human performance) for a problem, which if corrected, will 
prevent recurrence of that problem. 

The programmatic Root Causes of the RSG tube wear are: 

L Insufficient programmatic requirement to assure effective AVB contact force to 
prevent in-plane fluid elastic instability and random vibration and subsequent 
wear under high localized thermal-hydraulic conditions (steam quality (void 
fraction), flow velocity and hydro-dynamic pressure). 

Basis : The evaluation team concluded that the fundamental Root Cause for the 
in-plane FEI and the resulting tube-to-tube wear was the fact that in-plane FEI 
was not considered in the design of the SONGS RSGs. The fundamental reason 
for this lack of consideration was that industry practice and guidance, 
supported by the operating experience up to that time of U-bend type steam 
generators, indicated that the control out-of-plane FEI would prevent the 
occurrence of in-plane FEI. 

Likewise, the evaluation team concluded that the tube to AVB wear was caused 
by insufficient contact force under high localized thermal-hydraulic conditions, 
which was not recognized at the time of the design of the SONGS RSGs, and 
that the fundamental reasons for the ineffectiveness of the contact force were 
the established industry practice of minimizing the tube support gaps and 
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avoiding an excessive preload as well as other steps to control gap uniformity 
and parallelism. 

2. The design control process did not provide sufficient direction to assure that an 
evaluation of the need for an analysis of flow induced vibration of the retainer 
bar was performed and verified. 

Basis : The evaluation team concluded that the fundamental reason for the 
retainer bar FIV was the lack of clear direction in the MHI design procedures to 
require an evaluation to determine the different analyses and the level of 
analysis that were required for the RSG design in light of changes in the SONGS 
RSG design from previous MHI steam generator designs. 

5.9 Contributing Causes 

As used in this evaluation, "Contributing Causes" are defined as causes that by 
themselves would not create the problem but are important enough to be recognized 
as needing corrective action. Contributing causes are sometimes referred to as causal 
factors. Causal factors are those actions, conditions, or events that directly or indirectly 
influence the outcome of a situation or problem. The evaluation team closely 
evaluated the mechanistic causes and the design process for the potential existence of 
Contributing Causes. 



The programmatic Contributing Causes of the RSG tube wear are: 

r 



V J 
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6.0 Corrective Action Matrix 



Cause 


Corrective Action 


Due Date 


Root Cause 

^Insufficient 
programmatic 
requirement to assure 
effective AVB contact 
force to prevent 
in-plane fluid elastic 
instability and random 
vibration and 
subsequent wear 
under high localized 
thermal-hydraulic 
conditions (steam 
quality (void fraction), 
flow velocity and 
hydro-dynamic 
pressure). 


CAPR l:Revise Procedure 
5BBB60-N01 "Procedure for 
Controlling of the Design Activities" 
to require that the need for effective 
tube to AVB contact force under high 
localized thermal-hydraulic 
conditions(steam quality (void 
fraction), flow velocity and 
hydro-dynamic pressure) be 
addressed in all MHI SG designs. 


Completed 


CAPR l.o:Further revise Procedure 
5BBB60-N01 "Procedure for 
Controlling of the Design Activities" 
to require that sufficient contact 
force is assured under high localized 
thermal-hydraulic conditions (steam 
quality (void fraction) flow velocity 
and hydro-dynamic pressure), e.g., 
compare to the design parameters of 
previous successful MHI steam 
generator designs. 


11/15/2012 


CA I:Provide training for all Steam 
Generator Engineers (included new 
hires and continuing training) 
covering this event and the details 
concerning in-plane FEI and tube-AVB 
wear under high localized 
thermal-hydraulic conditions (steam 
quality (void fraction), flow velocity 
and hydro-dynamic pressure). 


Completed 
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Cause 


Corrective Action 


Due Date 


Root Cause 2:The 
design control process 
did not provide 
sufficient direction to 
assure that an 
evaluation of the need 
for an analysis of flow 
induced vibration of 
the retainer bar was 
performed and 
verified. 


CAPR 2;Revise procedure 
5BBB60-N01 "Procedure for 
Controlling of the Design Activities" 
to require that retainer bars and 
other steam generator parts subject 
to flow induced vibration be 
evaluated to determine the different 
analyses and the level of analysis that 
need to be performed to support the 
steam generator design. 


10/31/2012 


CA 2;Revise Engineer Training 
program (included new hires and 
continuing training) to include the 
necessary assessment for required 
analyses of each Steam Generator 
part subject to flow induced 
vibration. 


10/31/2012 
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Cause 



r 



Corrective Action 



Due Date 
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Cause 


Corrective Action 


Due Date 


Extent of Cause 


CA 6: Conduct a program design 
review for other SG design 
procedures and primary pressure 
boundary components (Reactor 
vessel, Core internals, Pressurizer, 
Reactor coolant piping, CRDMs) using 
senior engineers to determine if 
other design features have 
assumptions that are not 
programmatically captured and 
evaluated. 


3/31/2013 


CA 7: Reconfirm MHI steam 
generator designs using the 
procedure developed for Root Cause 

2. 


11/30/2012 

for SONGS SG design 


3/31/2013 

for OTHER SG designs 


CA 8: Reconfirm that the appropriate 
analyses were performed and that 
correct values were used as inputs 
for each thermal hydraulic analysis, 
vibration analysis, and wear analysis 
(FIT-MI, FIVATS, IVHET) in the design 
and fabrication processes of MHI 
steam generators. 


Completed for 
SONGS SG design 


10/31/2012 

for OTHER SG designs 


CA 9: Reconfirm that the computer 
validation was performed adequately 
for each thermal hydraulic analysis, 
vibration analysis, and wear analysis 
(FIT-HI, FIVATS, IVHET). 
*lf necessary, additional comparison 
to other validation methods shall be 
performed. 


Completed 
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Cause 


Corrective Action 


Due Date 


Effectiveness Review 


In accordance with MHI's QA 






program, "Corrective action reports" 






will be issued for all CAPRs and CAs 






and the confirmation of effectiveness 






of completed corrective actions will 






be performed by the Nuclear Plant 






Quality Assurance Section. 






Effectiveness reviews will be 






completed in six (6) months by 






verifying corrective actions for the 






addressed problems. 
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initial Unit 2 & 3 mid-cycle outage 






and SG inspections to determine the 






effectiveness of corrective actions. 






There is no evidence of : 
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• Additional tube to retainer bar wear 






(turbulence induced vibration 






(random vibration)) 






• Additional tube to AVB wear 






(turbulence induced vibration 






(random vibration)). 





Root Cause Analysis Report for tube wear identified in the Unit 2 and Unit 3 Steam 
Generators of San Onofre Nuclear Generating Station 



UES-20120254 Rev.O (32/64) Non-Proprietary 



7.0 Extent of Cause Evaluation 

The Root Causes were evaluated for the extent to which they would be applicable and 
present elsewhere in the MHI steam generator design process. 
The two Root Causes are: 

1. Insufficient programmatic requirement to assure effective AVB contact force to 
prevent in-plane fluid elastic instability and random vibration and subsequent 
wear under high localized thermal-hydraulic conditions (steam quality (void 
fraction), flow velocity and hydro-dynamic pressure). 

2. The design control process did not provide sufficient direction to assure that an 
evaluation of the need for an analysis of flow induced vibration of the retainer 
bar was performed and verified. 

Root Cause 1 is associated with the design program and procedures not capturing 
necessary design elements affecting the primary pressure boundary. MHI has different 
nuclear engineering sections responsible for different aspects of the primary pressure 
boundary design, and each section has its own controlling design programs and 
procedures. Therefore, the extent of cause applies to the SG design program and areas 
of design outside the SG design program that could impact the primary pressure 
boundary. Sections outside the SG program with design responsibility related to the 
primary pressure boundary include: 

a. Reactor Vessel 

b. Core internals 

c. Pressurizer 

d. Reactor coolant piping 

e. Control Rod Drive Mechanisms 

To address this extent of cause evaluation, each MHI engineering section will conduct 
a program and procedures review, based on what was learned from this event, to 
determine if there are other SG program elements or other primary components that 
rely on design assumptions that are not captured in the design program or procedures. 

For Root Cause 2, an analysis that should have been performed was not. Therefore, 
this extent of cause applies to other SG design analyses that should have been 
performed but were not. Because there is no controlling document that identifies what 
analyses should be performed for each component, CAPR 2 must be developed and 
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then a complete review of the different MHI SG project needs to be performed to 
confirm that all required analyses have been completed. 

8.0 Safety Culture Review 

A safety culture review was performed using the NRC's Inspection Manual Section 
IMC0310 COMPONENTS WITHIN THE CROSS-CUTTING AREAS and applying the 
guidance in that section to the Root and Contributing Causes identified in this report. 
The review examined all four safety culture areas, the thirteen cross-cutting and other 
area components, and the thirty-seven aspects comprised in those components. A 
summary table 1 that compares the identified Root and Contributing Causes with the 
requirements of each of the safety culture areas, components and aspects is provided 
below. 

As the table 1 shows, both Root Causes and all Contributing Causes are associated with 
aspect 6 (H.2(c)) of the "resources" component in the Human Performance Area. One 
Root Cause and all Contributing Causes are associated with aspect 2 (H.l(b)), of the 
"decision-making" component in the Human Performance Area. One Root Cause and 
all Contributing Causes are associated with aspect 4 (H.2(a)), of the "resources" 
component in the Human Performance area. Finally, one Root Cause and two of the 
Contributing Causes are associated with aspect 12 (H.4(c)) of the "work practices" 
component in the Human Performance Area. 

The component from the Human Performance Area applicable to the second Root 
Cause and the three Contributing Causes is aspect 6 (H.2(c)) of the "resources" 
component, which calls for complete, accurate and up-to-date design documentation, 
procedures, and work packages, and correct labeling of components. This aspect of the 
resources component was not satisfied because, while the decision making and the 
designs were properly documented, they were inaccurate in that they did not require 
analyses to evaluate the potential FIV of the retainer bars (Root Cause 2); 
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This component from the Human Performance Area is also associated with Root Cause 
1, in that the design procedures did not contain any requirement to assure effective 
AVB contact force. However, there is no safety culture related deficiency with 
respect to Root Cause 1 in that MHI was following accepted industry practices to 
design AVB and in fact sought to make its design more conservative than previous AVB 
designs. 

An aspect of a component from the Human Performance Area applicable to one of the 
Root Causes and the three Contributing Causes is aspect 2 (H.l(b)) of the 
"decision-making" component, which requires that conservative assumptions be used 
in the design. The design did not require analyses to evaluate the potential FIV of the 
retainer bars (Root Cause 2); 



v. J 
The discrepancies between the design and aspect 2 (H.l(b)) of the "decision-making" 

component also apply to aspect 4 (H.2(a)) of "resources" component. 

Finally, an aspect of a component from the Human Performance Area applicable to 
one Root Cause and two of the Contributing Causes is aspect 12 (H.4(c» of component 
4 ("work practices"), which requires that appropriate supervision and management 
oversight be applied to the design. While design activities were reviewed and 
confirmed by the design section the design supervision and review process failed to 
recognize that FIV analysis of the retainer bars was needed (Root Cause 2); 
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MHI has identified a number of corrective actions, which are being taken or will be 
completed in the near future, to address the safety culture discrepancies identified in 
this review. These corrective actions are described in Section 6.0 above. . The 
predominant safety culture aspect was determined to be H.2.(c) Work Documents 
because the decision making and work practices were not influenced by programmatic 
requirements. The H.2.(c) safety culture aspect has the associated corrective action to 
establish the programmatic requirements for both Root Causes and the Contributing 
Causes. 
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Table 1 Safety Culture Review - Cross Cutting Components and Aspect 



X: Not sufficient 



Safety Culture Area, 
Component, Aspect 


Root Cause 1 


Root Cause 2 




[ ) 


Insufficient programmatic requirement 
to assure effective AVB contact force to 
prevent in-ptane fluid elastic instability 
and random vibration and subsequent 
wear under high localized 
thermal-hydraulic conditions (steam 
quality (void fraction), flow velocity and 
hydro-dynamic pressure). 


The design control process did not provide 
sufficient direction to assure that an 
evaluation of the need for an analysis of 
flow induced vibration of the retainer bar 
was performed and verified. 




t ~\ 


Area 1. Human Performance (H) 


Component 1. Decision-Making 


Aspect 1. 

Risk significant decisions 
H.l(a) 


Sufficient - M Hi's AVB and tube bundle designs were reviewed and confirmed followed a decision-making process to evaluate and review the technical aspects of the design. 


Aspect 2. 
Conservative 
assumptions H.l(b) 


Sufficient - The AVB design decision was 
based on a FIT-HI analysis which had a 
built in safety margin and assumed one 
inactive support as an additional measure 
of conservatism additionally M Hi's design 
had more AVBs than previous designs. 


X 


Not sufficient - The engineer 
responsible for the retainer bar design 
did not recognize the need to analyze 
the retainer bar for potential flow 
induced vibration 






Aspect 3. 

Timely communication 
H.l(c) 


Sufficient - The decisions of the AVB and SG team were documented and distributed to the team members in a timely manner. 


Component 2. Resources 


Aspect 4. 

Managing maintenance 
H.2{a) 


Sufficient - The FIT-III analysis had a built 
in safety margin and assumed that one 
inactive support as an additional measure 
of conservatism. 


X 


Not sufficient - The engineer did not 
recognize the need to analyze the 
retainer bars for potential FIV. 
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Safety Culture Area, 
Component, Aspect 


Root Cause 1 


Root Cause 2 






Aspect 5. 
Training and 
qualification personnel 
H.2(b) 


X 


Not sufficient - While the design section included experts in SG design and manufacture, however procedure for training program was not sufficient because the training materials and procedures were inadequate. 


Aspect 6. 

Work documents H.2(c) 


Not deficient The decision making and 
design were documented, but the design 
procedures did not include a requirement 
to prevent in-plane FEI and random 
vibration related wear under high 
localized thermal-hydraulic conditions. 
There was no programmatic requirement 
to prevent in-plane FEI and random 
vibration, but MHI sought to make the 
AVB design more conservative than 
previous designs so no safety culture 
deficiency is found. A corrective action is 
nevertheless provided to address this new 
understanding based on the tube wear 
observed at SONGS. 


X 


Not sufficient - The decision making and 
design were documented, but the 
design procedures did not include a 
requirement to evaluate the retainer 
bars for potential FIV. The predominant 
safety culture aspect was determined 
to be H.2.(c) Work Documents because 
there was no programmatic 
requirement to influence the engineer. 
The H.2.(c} safety culture aspect has the 
associated corrective action to establish 
the programmatic requirement to 
evaluate for the need for an FIV analysis 




k J 


Aspect 7. 

Facilities and Equipment 
H.2(d} 


Sufficient - The SG design section was provided with adequate facilities and other resources to conduct design review meetings and decision-making. 


Component 3. Work Control 


Aspect 8. 

Work planning H,3(a) 
Aspect 9. 

Work coordinationH.3(b) 


Not applicable - Aspects 8 and 9 are not applicable because they address work in the plant and coordination of removal of safety systems during plant maintenance. 


Component 4. Work Practices 


Aspect 10. 
Error prevention 
techniques H.4(a) 


Sufficient - Design activities were established in compliance with QA programs to prevent error and personnel followed appropriate procedures. 


Aspect 11. 

Procedure compliance 
H.4(b} 


Sufficient - MHI's corrective action program governed the design process. Additionally the design section decisions were made pursuant to decision making procedures. 
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Safety Culture Area, 
Component, Aspect 


Root Cause 1 


Root Cause 2 


r ] 
i j 


Aspect 12. 
Supervision and 
management oversight 
H.4(c) 


Sufficient -MHI's SG design activities were 
reviewed and confirmed by the design 
section at design review and technical 
review meetings. 


X 


Not sufficient - The need for a FIV 
analysis of retainer bar was not 
detected in the design review process. 




Area 2. Problem Identification and Resolution (P) 


Component 5. Corrective Action Program 


Aspect 13. 

Risk-based identification 
threshold P.l(a) 


Sufficient - MHI's corrective action program governed the design process. 


Aspect 14. 

Trending program P. 1(b) 


Sufficient - MHI's corrective action program includes trend based assessments. 


Aspect 15. 

Cause evaluations P. 1(c) 


Sufficient - MHI's corrective action program includes Root Cause and apparent cause assessments. 


Aspect 16. 

Corrective actions P. 1(d) 


Sufficient - No unresolved corrective actions were at issue. 


Aspect 17. 

Alternative processes 
P.l(e) 


Sufficient - MHI has alternative programs in addition to its regular reporting program. 


Component 6. Operating Experience 


Aspect 18. 
Systematic process 
P.2(a> 


Sufficient - MHI investigated operating experience with U-bend tube degradation using INPO, NRC and NPE data bases, and communicated internally in a timely manner. 


Aspect 19. 

Process changes P. 2(b) 


Sufficient - MHI conducted benchmarking and concluded that the SONGS RSG was designed to minimize the potential for tube wear by providing more support points with shorter spans in the U-bend region along with effective zero 
tube-to-AV8 gaps during SG operation. 


Component 7. Self- and Independent Assessments 


Aspect 20. 

Nature of assessments 
P.3(a) 


Sufficient - MHI periodically and appropriately conducted self -assessments. 
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Component, Aspect 


Root Cause 1 


Root Cause 2 


[ ] 


Aspect 21. 

Tracking and trending 
P.3{b> 


Sufficient - MHI periodically and appropriately conducted self-assessment. 


Aspect 22. 
Coordination and 
communication P. 3(c) 


Sufficient - MHI coordinated and communicated result from self-assessment to affect personnel and took appropriate corrective actions. 


Area 3. Safety Conscious Work Environment (S) 


Component 8. Environment for Raising Concerns 


Aspect 23. 
Free and open 
information exchange 
5.1(a) 


Sufficient - The SG design team and AVB design team encouraged discussions of safety issues and openly exchanged information on design alternatives 


Aspect 24. 
Alternate processes 
5.1(b) 


Sufficient - MHI has alternative programs for raising safety concerns in confidence. 


Component 9. Preventing, Detecting, and Mitigating Perceptions of Retaliation 


Aspect 25. 
Training 5.2(a) 


Sufficient - There were no claims of harassment by SG team members. 


Aspect 26. 
Investigation S.2(b) 


Sufficient - There were no claims of retaliation by SG team members. 


Aspect 27. 
Chilling effect S.2{cJ 


Sufficient - MHI appropriately considers chilling effect. No disciplinary actions were taken. 


Area 4. Other Safety Culture Components (0) 


Component 10 .Accountability 


Aspect 28. 

Alignment of safety and 
rewards 0.1(a) 


Sufficient - Accountability for SG design decisions was clearly understood within MHI. 


Aspect 29. 

Reinforcement 0.1(b) 


Sufficient - Management reinforced safety standards. 


Aspect 30. 
Safety focus 0.1(c) 


Sufficient - MHI demonstrated safety focus, review of meeting minutes indicates focus of SG design team was to come up with design with appropriate margin which demonstrated focus on safety. 
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Safety Culture Area, 
Component, Aspect 


Root Cause 1 


Root Cause 2 


( ] 


Component 11. Continuous (earning environment 


Aspect 31. 

Training and knowledge 
0.2(a) 


Sufficient - SG design team was trained on SG design continuously. 


Aspect 32. 

Internal and external 

learning 0.2(b) 


Sufficient - SG design team was trained on SG design continuously and transfer of knowledge was provided by internal experts, however the issue of effective AVB to contact force was not known within the industry so training could 
not be effective. 


Components. Organizational change management 


Aspect 33. 

Organizational change 
management 0.3 


Sufficient - MHI management used systematic process and evaluated of impacts of decisions when organization was changed. 


Components. Safety policies 


Aspect 34. 

Raising concerns0.4(a) 


Sufficient - MHI has appropriate policies which required reinforce to raise safety concern. 


Aspect 35. 

Safety policy training 

0.4(b) 


Sufficient - MHI has appropriate policy training to raise individual safety concern. 


Aspect 36. 
Decisions consistent 
with safetypriority0.4(c) 


Sufficient - Decisions related to SG design were consistent with MHI policies. 


Aspect 37. 

Top management 

commitment 0.4(d) 


Sufficient - Top management communicated need for safe SG design as issue of effective AVB to contact force was not known within the industry so management communication could not be effective. 
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Cause-effect analysis for the tube to tube wear 



(Legend) 



| : Phenomenon 



Tube Failure 

i 



In-plane FEI'Note 1 



High localized TH condition 



Low Natural Frequency (in-plane) 



High Velocity 



Underestimated the velocity because 
inappropriate gap was assigned as 
vibration analysis input 



Localized high steam 
quality (void fraction) 



Inadequate support 
in-plane direction 
(insufficient tube to AVB 
contact force) 



Insufficient action to decrease 
steam quality (void fraction) 



T/H condition was judged acceptable by FIV 
analysis (SR<1 and no excessive wear) and 
dry out evaluation 




"B" is underlying cause of "A" 
Cause 

Contributing Cause 
Root Cause 



No AVB for 
in-plane 



Design element to 
avoid excessive 
preload (contact force) 
between AVB and 
tube 



Other potential design changes 
considered to decrease steam 
quality (void fraction) but they 
had unacceptable outcomes 



T/H analysis (FIT- III) did not indicate 
the necessity to reduce the high steam 
quality (void fraction) 



Large bending 
radius of U-tube 



Large SG 



Replacement from 
OSG 



Insufficient 
knowledge of the 
relationship 
between high 
steam quality (void 
fraction) and 
in-plane FEI 



To maintain 
mechanical 
damping by the 
sliding of tube 
along AVB 



Did not 
recognize 
need for 
contact force 
to control 
in-plane FEI 
'Note 2 



Root Causes are defined as the basic reasons (e.g., hardware, 
process, or human performance) for a problem, which if 
corrected, will prevent recurrence of that problem. 
Contributing Causes are defined as causes that by themselves 
would not create the problem but are important enough to be 
recognized as needing corrective action. Contributing Causes 
are sometimes referred to as causal factors. Causal factors are 
those actions, conditions, or events that directly or indirectly 
influence the outcome of a situation or problem. 



MHI relied on the 
performance of FIT-III 
based on the past 
successful experiences 
in triangular tube 
configuration SGs 



Root Cause (1) 

Insufficient programmatic 
requirement to assure effective 
AVB contact force to prevent 
in-plane fluid elastic 
instability and random 
vibration and subsequent wear 
under high localized 
thermal -hydraulic conditions 
(steam quality (void fraction), 
flow velocity and 
hydro-dynamic pressure). 



Satisfy design 
specification 
not to exceed 
7% ding 
indications with 
ECT 'Note 3 



MHI SONGS RSG 
design did not consider 
the phenomenon of 
in-plane FEI because 
contemporary knowfedge 
and industry U-bend SG 
operation experience did 
not indicate a need to 
consider in-plane FEI 



No previous 
SG 

experience 
showed need 
for in-plane 
FEI analysis 



'Note 1: Each SG on unit 3 had about 160 
tubes with TTW. Only 2 tubes in one unit 2 SG 
had TTW. This difference was caused by 
manufacturing and fabrication improvements 
impiemented for unit 3. These improvements 
resulted in lower tube to AVB contact force in 
unit 3 SGs when compared to unit 2 SGs. 

'Note 2: At the time of design a narrow gap 
between the AVB and tube was believed to 
achieve effective support condition. 

'Note 3: To consider identifying specific 
customer specification but also note that MHI 
evaluated and found such 
specification/requirement acceptable based 
on experiences. 



Root Cause Analysis Report for tube wear identified in the Unit 2 and Unit 3 Steam Generators of San Onofre Nuclear Generating Station 



Attachment-1 UES-20120254 Rev.O (42/64) |Non-Proprieta 



i 



Cause-effect analysis for the tube to AVB wear 



Tube to AVB wear 



(Legend) 



Random vibration 



[ ] : Phenomenon 



IZHZ 



High localized TH condition 



High Velocity 



Inadequate support (insufficient tube to 
AVB contact force) 



Underestimated the velocity because 
inappropriate gap was assigned as 
vibration analysis input 



Localized high steam 
quality (void fraction) 



Insufficient action to decrease 
steam quality (void fraction) 



"B" is underlying cause of "A" 
Cause 

Contributing Cause 
Root Cause 



Design element to avoid excessive preload 
(contact force) between AVB and tube 



T/H condition was judged 
acceptable by F1V analysis (SR<1 
and no excessive wear) and dry out 
evaluation 



Other potential design changes 
considered to decrease steam 
quality (void fraction) but they 
had unacceptable outcomes 



T/H analysis (FIT- lll)did not indicate 
the necessity to reduce the high steam 
quality (void fraction) 



Two inactive supports were 
deemed to be a sufficiently 
conservative assumption for 
evaluation of vibration wear 
estimation even under high 
steam quality conditions 



To maintain 
mechanical damping 
by the sliding of tube 
along AVB 



Did not recognize 
need for contact 
force to control 
random vibration 
*Note 1 



Satisfy design 
specification not to 
exceed 7% ding 
indications with ECT 
'Note 2 



Root Causes are defined as the basic reasons (e.g., hardware, 
process, or human performance) for a problem, which if 
corrected, will prevent recurrence of that problem. 
Contributing Causes are defined as causes that by themselves 
would not create the problem but are important enough to be 
recognized as needing corrective action. Contributing Causes 
are sometimes referred to as causal factors. Causal factors are 
those actions, conditions, or events that directly or indirectly 
influence the outcome of a situation or problem. 



MHI relied on the 
performance of FIT-III 
based on the past 
successful experiences 
in triangular tube 
configuration SGs 



Root Cause (1) 

Insufficient programmatic 
requirement to assure effective 
AVB contact force to prevent 
in-plane fluid elastic instability 
and random vibration and 
subsequent wear under high 
localized thermal -hydraulic 
conditions (steam quality (void 
fraction), flow velocity and 
hydro-dynamic pressure). 



MHI SONGS RSG design did not 
consider sufficient AVB contact force 
to pievent the random vibration 
under high steam quality (void 
fraction) condition because 
contemporary knowledge and 
industry U-bend SG operation 
experience did not indicate a need 
to consider such phenomenon 
'Note 1 



'Note 1: At the time of design a narrow 
gap between the AVB and tube and low 
contact force was believed to achieve 
effective support condition. 

*Note 2: To consider identifying specific 
customer specification but also note that 
MHI evaluated and found such 
specification/requirement acceptable 
based on experiences. 
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Cause-effect analysis for the retainer bar to tube wear 



Retainer bar to tube wear 



(Legend) 



( ] : Phenomenon 



Random vibration 



□ 



□ 



"B" is underlying cause of "A" 
Cause 
: Root Cause 



Design allowed vibration 



Bar was exposed to turbulent 
flow 



The bar length was 
extended 



Needs to capture larger 
number of tubes 





Bar was designed 






thinner 








\ 


f 


Allowable space between 


tubes 





FIV analysis was not performed 



No assessment of 
need for FIV analysis 
performed for retainer 
bar 



RSG design elements 



Senior engineer did not 
identify the need for FIV 
analysis during the design 
verification review 



Root Causes are defined as the basic reasons (e.g., hardware, 
process, or human performance) for a problem, which if corrected, 
will prevent recurrence of that problem. 



Based on past 
success, engineering 
did not recognize the 
need to perform 
additional analysis 



Previous designs did 
not perform FIV for 
retainer bar 



Root Cause (2) 

The design control process did not provide 
sufficient direction to assure that an evaluation of 
the need for an analysis of flow induced vibration 
of the retainer bar was performed and verified. 
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Barrier analysis 

A barrier analysis for the SONGS RSGs was performed with respect to the items listed 
below. 



• Design 

• Fabrication 

• Post Installation testing/monitoring 

• Post Operational Inspection 

The barrier analysis was developed to assess the barriers at each of the major stages of 
the steam generator replacement program. The two primary barriers assessed at each 
stage included training/ qualification of personnel and procedures. As shown in the 
Barrier analysis table, procedures and training / qualification were lacking for the three 
wear mechanisms evaluated (tube to tube, retainer bar to tube, and tube to AVB). 
There were no issues identified with the fabrication process so there were no failed 
barriers. The results of the barrier analysis support the cause-effect analysis. 
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Consequ 
ence 


Barrier 


Outcome 


Evaluation 


Tube to 

Tube 

wear 


Design 


Training / 
Qualificati 
on 


In-plane FEI 


Not 
Effective 


The design section procedure did not contain guidance on in-plane FEI. As a result, training programs did 
not cover the phenomenon of in-plane FEI. This will be addressed with CA 1 in the Corrective action matrix. 








TH model 
FIV analysis 


Not 
Effective 


f ~\ 

J 






Procedures 


In-plane FEI 


Not 
Effective 


Analyses were not performed because there was no consideration of this phenomenon mentioned in the 
procedure. This will be addressed with CAPR 1 in the Corrective action matrix. 








TH model 
FIV analysis 


Not 
Effective 


FIT-III predicted a lower velocity due to use of inappropriate gap value. This will be addressed with CA 3 in 
the Corrective action matrix. 






Supervisio 
n 


In-plane FEI 


Not 
Effective 


Supervisors used same procedures and received the same training as design engineers. 

Fvprv 3 months an Executive Oversight Mpptine was held Dpoutv Head of MHI and DeDartment Managers 
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participated. However, there were no questions related to in-plane FEI because it was not considered under 
MHI procedure or industry practice. This will be addressed with CAPR 1 and CA 1 in the Corrective action 
matrix. 








TH model 
FIV analysis 


Potentially 
Not 

Effective 


FIT-III output indicated higher steam quality (void fraction) than previous SG designs. However, the senior 
engineer did not consider the potential adverse effects of the higher steam quality (void fraction). 
This will be addressed with CA lin the Corrective action matrix. 
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Retainer 
to tube 


Design 


Training 


Required 
Analyses 


Not 
Effective 


Necessary analyses for each component were selected based on engineering judgment and past success. 
Training was insufficient. 

This will be addressed with CAPR2in the Corrective action matrix. 


bar wear 




Procedures 


Required 
Analyses 


Not 
Effective 


There was no requirement to confirm the consideration of a FIV analysis for changes made to a component 
in the flow stream. 

This will be addressed with CA 2in the Corrective action matrix. 






Supervision 


Not 
Effective 


Based on past successful experience, engineering did not recognize the need to perform additional analysis 
for the retainer bars. The senior engineer did not identify the need for FIV analysis during the design 
verification review. 

This will be addressed with CA 2in the Corrective action matrix. 


Tube to 
AVB wear 


Design 


Training 


Contact force 
under high 
steam quality 


Not 
Effective 


SG design training does not discuss contact force as a control mechanism to address vibration related wear 

under high steam quality (void fraction) condition. 

This will be addressed with CA lin the Corrective action matrix. 






Procedur 
es 


Contact force 
under high 
steam quality 


Not 
Effective 


SG design procedures do not mention AVB contact force as a control mechanism to address vibration 
related wear under high steam quality (void fraction) condition. 
This will be addressed with CAPRlin the Corrective action matrix. 


Consequence all 


Fabrication 


Not 
applicable 


The SGs were fabricated as intended. For unit 2 it was done using the normal fabrication process. For unit 3 
it required divider plate failure repair. There were no causes identified associated with fabrication deviation 
from the design. 
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Change analysis 



For the SONGS RSGs, a change analysis was performed in two stages. The first stage 
compared the SONGS SG design to previous MHI SG designs for the triangular tube 
configuration. MHI had previously performed three steam generator designs using a 
triangular tube configuration. The second stage compared the SONGS RSGs to the 
previous SONGS SG design (Combustion Engineering type design). Only the most 
significant changes are included in this analysis. 



The change analysis results are set out below. 
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(3) Identification of the changes from previous SG designs led to the recogn ition that 
the RSG design deserved close scrutiny. MHI considered the changes in the SONGS 
design from previous steam generator designs and compared the basic design 
parameters of the SONGs RSGs (e.g., heat transfer area, circulation ratio, steam 
pressure, etc.) with other steam generator designs. Further, as part of the 
development of the SONGS RSG design, MHI conducted a detailed comparison 
between its proposed AVB support for the tubes in the U-bend region and that of a 
comparison plant of similar design. A special AVB team was formed and included 
industry experts to conduct an extensive design review process in 2005 / 2006 to optimize 
the U-bend design and address the technical issues. The team concluded that the 
SONGS design was significantly more conservative than previous designs in 
addressing U-bend tube vibration and wear. 

Also MHI and SCE recognized that the SONGS RSG steam quality (void fraction) was 
high and MHI performed feasibility studies of different methods to decrease it. 
Several design adjustments were made to reduce the steam quality (void fraction) 
but the effects were small. Design measures to reduce the steam quality (void 
fraction) by a greater amount were considered, but these changes had 
unacceptable consequences and MHI and SCE agreed not to implement them. It 
was concluded that the final design was optimal based on the overall RSG design 
requirements and constraints. These included physical and other constraints on the 
RSG design in order to assure compliance with the provisions of 10 C.F.R. §50.59. 
Thus, MHI did compare the SONGs RSG design with previous steam generator 
designs, and in particular did a detailed evaluation of different options of the AVB 
design taking into account other large steam generator designs. 
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RCA charter 

Title: Root Cause Analysis Report for tube wear identified in the Unit 2 and Unit 3 Steam 
Generators of San Onofre Nuclear Generating Station 



Management Sponsor: 
Team: f 



) 



Problem Statement: 

(1) Requirement 

No Primary-to-Secondary Leakage due to Defects in any of the RSG Units for the 
duration of the Warranty Period, (per 17.2.3 of General T&C with EMS) 

(2) Deviation 

Unit 3 SG-B (SCE SG088) experienced tube leakage during operation and failure of eight 
tubes during in-situ pressure testing. (Both due to Defects) 

(3) Consequences (For MHI) 



Timeline and Deliverables: 

• RCA Team Assigned : March 23, 2012 

• Problem Statement committed : March 23, 2012 

• Prepare begun: March 26, 2012 

• DRAFT Cause-effect analysis : Aprill3, 2012 

• DRAFT RCA Summary : JulyS, 2012 

• Review RCA Summary : July7, 2012 

• DRAFT RCA Report : July20, 2012 

• Review Revised RCA Summary : August30, 2012 

• Review RCA report: September 6 and 11, 2012 

• RCA Due Date: Octoberl2, 2012 



Root Cause Analysis Report for tube wear identified in the Unit 2 and Unit 3 Steam 
Generators of San Onofre Nuclear Generating Station 



( 



Attachment-5 UES-20120254 Rev.O (55/64) |Non-Proprietary| 



] 



Time line 



Design Procedure 
(BKA6O-N01) 
Time Line 



Rsv.O. 5731 
(Replaced -|- 
with 5BBB60- 
NOt) 



Rev.17. 2/12 
Rev.16. 2/5 



Rev.15. 12/2~ 
Rev.14. 10/23 " 
Rev.13. 10/19" 



Rev. 12. 4/26 
Rev. 11. 4/1 2 



Rev. 7. 7/13 
Rev 6, 5/17 



Rv.4. 10/22 
Rev.3. a/23 



Rev.5. 7/3 
Rev.4. 3/21 



Rev.3. 4/4 
Rev.2. 2/27 



Rev.1. 8/22 



Rev.O. 4/26 



Rev.3. 2/17 
Rev.2. 1/19 

Rev.1. 11/25 

Rev.O. 9/30 



Rev.2. 1/19 
Rev.1. 9/20 



Rev.O. 2/28 



Rev.9. 8/29 
Rev.8. 4/7 



Rev.7. 2/27 
Rev.6. 1/23 



Rev.5. 10/21 
Rev.4. 9/29 



Rev.2. 6/6 
Rev.1 4/29 
Rev.O. 3/9 



Rev,7. 3/12 
Rev.6. 1/15 



Rev.5. 6/11 



Rev.4. 3/24 
Rev.3. 2/29 



Rev.5. 7/14 
Rev.4. 6/12 



Rev.2. 2/3 

Rev.1. 11/14 
Rev.O. 9/5 



Rev.2. 4/29 

Rev.1. 1/7 
Rev.O. 11/30 



Thermal and 

Hydraulic 

Parameters 

LS-04GA503 

Superseded by 

LSO4GA510 

Rev.1 



Deatgn of Analytical Water Level Data for 

Anti-Vibration Report of Controllability Licensing 

Bar AVB Analysis Support 

L5-04GA428 Assembly L5-04GA519 Analysis 

L5-MGA419 L5-04GA511 



Design Design 

Report for the Report of 

Tube Sheet Tube 

Region L5-04GA418 
L5-04GA401 



Root Cause Analysis Report for tube wear identified in the Unit 2 and Unit 3 Steam 
Generators of San Onofre Nuclear Generating Station 



( 



Attachment-5 UES-20120254 Rev.O (56/64) |Non-Proprietary| 



) 



Design Procedure 
(BKA60-N01) 
Time Line 



Rev.O. 5/31 _ _ 
(Replaced — 
with 5BBB60. 
N01) 



Rev.lB. 11/24 



Rev.17. 2/12 
Rev.16, 2/5 



Rev.15. 12/21 
Rev.14. 10/23" 
Rev.13. 10/19' 



Rev. 12, 4/26 
Rev. 11. 4/1 2 



Rev,9. 10717 
Rev.8. 9/9 



Rev.7. 7/13 
Rev.6, 5/17 



Rev.2. 1/19 
Rev.1. 11/11 
Rev.O. 9/30 



Rev.3. 7/29 
Rev.2. 5/17 
Rev. 1.3/9 
Rev.O. 1/21 



Rev. 1.9/9 
Rev.O. 6/27 



Rev.7. 6/4 
Rev.6. 4/4 
Rev.5. 2/29 



Rev.4. 10/7 
Rev.3. 917 

Rev.2. 6/14 

Rev.1 . 3/23 
Rev.O. 1/19 



Rev.3. 8/4 



Rev.2. 2/28 



Rev.9. 9/25 
Rev 8. 8/20 



Rev.6. 4/2 
Rev.5. 2/13 



Rev.4. 12/12 
Rev.3. 11/17 



Rev.2. 10/5 
Rev.1. 8/12 
Rev.O. 6/27 



Rev.O. 2/24 



Rev.4. 5/12 
Rev.3. 4/19 
Rev.2. 3/24 

Rev.1. 1/20 

Rev.O. 11/18 



Rv4. 10/22 
Rev.3, 8/23 



Material 
Selection 
Report for 
Anti-Vibration 
Bar 

L5-04GA224 



Material 
Selection 
Report for 
Heat Transfer 
Tubing 
L5-04GA208 



Material 
Selection 
Report for 
Tube Support 
Plate 

L5-04GA223 



Design 
Report of 
Tube Support 
Plate and 
Stay Rod 
L5-04GA41 1 



Basic Sizing Design of Qualification 

Calculation Tube Support Report of 

Report Plate AVB End Cap 

L5-04GA421 L5-04GA425 L5-04GA429 



Root Cause Analysis Report for tube wear identified in the Unit 2 and Unit 3 Steam 
Generators of San Onofre Nuclear Generating Station 



[ , ) 

Attachment-5 UES-20120254 Rev.O (57/64) [Non-Proprietaryl 



Design Procedure 
(BKA60-N01) 
Time Line 



Rev 0. 5/31 _ _ 
{Replaced - - 
with 58BB60 



Rev.18. 11/24 



Rev.17, 2/12 
Rev 16, 2/5 



Rev 15. 12/2T^ 
Rev 14. 10/23" 
Rev 13. 10/19" 



Rev. 12. 4/26 
Rev 11, 4/12 



Rev.5. 11/12 
Rev.4. 10/28 



Rev. 3. 8/29 



Rev.2. 3/31 
Rev.1. 1/31 



Rev.O. 8/3 



Rev.7, 7/13 
Rev 6. 5/17 



Rv4, 10/22 
Rev 3, 8/23 



Rev.2. 10/18 



Rev.1. 6/13 



Rev. 1.8/22 



Rev.O. 11/11 Rev.O. 11/11 IRev.O. 11/11 IRev.O. 11/11 Rev.O. 11/11 Rev.O. 11/11 



Rev.2. 10/18 



Rev.1. 6/13 



Rev.2. 10/18 
Rev.1. 6/13 



Thermal and 

Hydraulic 

Parametric 

Calculations 

L5-04GA510 



Anti- Vibration 
Bar Assembly 
1/9 

L5-04FU111 



Anti-Vibration 
Bar Assembly 
2/9 

L5-04FU112 



Anti-Vibration 
Bar Assembly 
3/9 

L5-04FU113 



Anti-Vibration 
Bar Assembly 
4/9 

L5-04FU114 



Anti-Vibration 
Bar Assembly 
5/9 

L5-04FU115 



Anti-Vibration 
Bar Assembly 
6/9 

L5-04FU116 



Root Cause Analysis Report for tube wear identified in the Unit 2 and Unit 3 Steam 
Generators of San Onofre Nuclear Generating Station 



( , J 

Attachment-5 UES-20120254 Rev.O (58/64) |Non-Proprietaryj 



Design Procedure 
(BKA60-N01) 
Time Line 



Rev.O. 5/31 
(Replaced 
with 5BBB60- 
N01) 



Rev. 17, 2/12 
Rev 16, 2/5 



Rev. 15, 12/21 
Rev.14. 10/23 " 
Rev. 13, 10/19" 



Rev.9, 10/17 
Rev. 8, 9/9 



Rev. 7, 7/13 
Rev. 6, 5/17 



Rev.9. 3/9 

Rev.8. 12/8 
Rev.7. 12/5 
Rev.6. 11/9 
Rev.5. 10/18 
Rev.4. 9/26 
Rev. 3. 8/24 

Rev.2. 6/13 



Rev.1. 2/28 
Rev.O. 11/11 



Dimesion change of 

Retainer Bar 

Stress Engineer sign off 



Rev.3. 4/6 



Rev.2. 2/17 
Rev.1. 1/19 



Rev.O. 11/11 



Rev.1. 1/19 
Rev.O. 11/11 



Rev.3. 9/4 



Rev.2. 12/2 

Rev.1. 8/12 
Rev.O. 5/27 



Rv.4. 10/22 
Rev 3, 8/23 



Rev.3. 8/10 
Rev.2. 7/24 
Rev.1. 6/8 



Rev.O. 2/15 



Rev.3. 8/10 
Rev.2. 7/24 
Rev.1. 6/8 



Rev.O. 2/15 IRev.O. 2/15 



Anti-Vibration Anti-Vibration Anti-Vibration Tube Support General 

Bar Assembly BarAssembly Bar Assembly Plate Assembly 1/3 

7/9 8/9 9/9 Assembly 3/3 KAZ-04-0021 

L5-04FU117 L5-04FU118 L5-04FU119 L5-04FU108 



General General 
Assembly 2/3 Assembly 3/3 
KAZ-04-0022 KAZ-04-O023 



Root Cause Analysis Report for tube wear identified in the Unit 2 and Unit 3 Steam 
Generators of San Onofre Nuclear Generating Station 



Attachment-5 UES-20120254 



( ) 

Rev.O (59/64) |Non-Proprietary| 



Design 
procedure 
(BKA60-N01) -(_ 
Trm Line 



Rev.5. 6/18 
Rev.4. 5/22 



Rev.O. 5/31 " 
(Replaced with" 
5BBB6O-N01) _ _ 



Rev.4. 6/20 
Rev.3. 5/11 
Rev.2. 4/5 
Rev.1. 3/16 
Rev.O. 2/13 



Rev.9. 9/27 
Rev.8, 9/14 
Rev.7, 8/30 
Rev.6. 8/10 
Rev.5. 7/17 
Rev.4. 7/13 
Rev.3. 6/27 
Rev.2. 5/21 
Rev.1. 4/27 
Rev.O. 3/30 



Rev.3, 9/27 
Rev.2, 8/31 
Rev.1, 7/27 
Rev.O. 7/4 



10/12 



3/23 



Rev. 17. 2/12 ~ 
Rev.16. 2/5 ' 



Rev,9. 8/4 

Rev.8. 6/16 

Rev.7. 4/19 

Rev6 2/13 

Rev.5. 12/21 

Rev.4. 10/21 
Rev.3. 9/26 
Rev.2 8/22 

Rev.1. 5/17 



Rev.O. 10/15 



Rev.3. 12/11 



R-Violette. M.J.Pettigrew. and N.W.Mureithi 
"Ftuidelastic Instability of an Array of Tubes Preferentially 
Flexible in the Flow Direction Subjected to Two-Phase 
Cross Flow" 

ASME Journal of Pressure Vessel Technology Vol.1 28 
Feb.2006 




V P Janzen, EG Hagberg, M.J.Pettigre 
and C E. Taylor 

Fluidelastic instability and Work-Rate 
Measurements of Steam-Generator 
U- Tubes in Air-Water Cross -Flow" 
ASME Journal of Pressure Vessel 
Technology Vol.127 
Feb 2005 



M. K. Au-Yang. 
"Flow- Induced Vibration 
of Powerand Process 
Plant Components" 
(Jul. 2001 




10/2 
9/11 



3/13 



12,5 

10/17 
9/14 
6717 
7/11 

5/23 



JSME Guideline "Fluid-elastic Vibration 
Evaluation of U-bend Tube in SGs" 
Mar 2002 




RBptacement Evafciatjonol Ratarier Bar Tubeweerof Vakrjty of Use 

Steam Tube Vbrauon Tube Wear Unit- 3 RSG - of the FTT-a 

Generators L5-04GA504 Rrifjorl Technical RbsuNs tuning 

Project L5-04GA561 Evakjauon Desrjn 

bxecuuonRan Report L5-04GA591 

L5-MGB001 L5-04GA564 



Acbvrty period M+SCE Design 

of the RCA team Review Meeung 
for AVB Design 
and Fabrication 



Root Cause Analysis Report for tube wear identified in the Unit 2 and Unit 3 Steam 
Generators of San Onofre Nuclear Generating Station 



Attachment-6 UES-20120254 Rev.O (60/64) |Non-Proprietary| 



Interview list 



r 



v 



Root Cause Analysis Report for tube wear identified in the Unit 2 and Unit 3 Steam 
Generators of San Onofre Nuclear Generating Station 



( , ] 

Attachment-6 UES-20120254 Rev.O (61/64) |Non-Proprietary| 



Root Cause Analysis Report for tube wear identified in the Unit 2 and Unit 3 Steam 
Generators of San Onofre Nuclear Generating Station 



( ) 

Attachment-7 UES-20120254 Rev.O (62/64) |Non-Proprietary| 



Reference documents 

1. 5BB60-N06Rev.l 

"ASME Code Job Procedure for Preparation of Design Reports", 5/30/2012 

2. 5BBB60-N01Rev.O 

"Procedure for Controlling of the Design Activities", 5/31/2011 

3. KAS-20050202 Rev.l 

"FIT-IN Code Description Note (Code User's Manual)", 11/18/2005 

4. L5-04GA504 Rev.5 

"Evaluation of Tube Vibration", 6/18/2012 

5. L5-04GA588 draft 

"Summary of Technical Evaluation" 

6. L5-04GA561 Rev.4 

"Retainer Bar Tube Wear Report", 6/20/2012 

7. L5-04GA564 Rev.9 

"Tube wear of Unit-3 RSG - Technical Evaluation Report", 9/27/2012 

8. L5-04GA591Rev.3 

"Validity of Use of the FIT-IN Results during Design", 9/27/2012 

9. L5-04GA428 Rev.5 

"Design of Anti-Vibration Bar", 7/3/2008 

10. L5-04GA419 Rev.4 

"Analytical Report of AVB Assembly", 7/3/2008 

11. L5-04GA519Rev.2 

"Water Level Controllability Analysis", 1/19/2007 

12. L5-04GA511 Rev.9 

"Data for Licensing Support Analysis", 8/29/2008 

13. L5-04GA401 Rev.7 

"Design Report for the Tube Sheet Region", 3/12/2010 

14. L5-04GA418 Rev.5 

"Design Report of Tube", 7/14/2008 

15. L5-04GA224 Rev.2 

"Material Selection Report for Anti-Vibration Bar", 1/19 2006 

16. L5-04GA208 Rev.3 

"Material Selection Report for Heat Transfer Tubing",7/29/2005 

17. L5-04GA223 Rev.l 

"Material Selection Report for Tube Support Plate", 9/9/2005 

18. L5-04GA411 Rev.7 

"Design Report of Tube Support Plate and Stay Rod", 6/4/2008 

19. L5-04GA421 Rev.4 

"Basic Sizing Calculation Report", 1/25/2010 

20. L5-04GA425 Rev.9 

"Design of Tube Support Plate", 9/25/2008 

21. L5-04GA429 Rev.4 

"Qualification Report of AVB End Cap", 5/12/2006 



Root Cause Analysis Report for tube wear identified in the Unit 2 and Unit 3 Steam 
Generators of San Onofre Nuclear Generating Station 



( , ) 

Attachment-7 UES-20120254 Rev.O (63/64) |Non-Proprietary| 



22. L5-04GA510 Rev.5 

"Thermal and Hydraulic Parametric Calculations", 11/12/2008 

23. L5-04FU111 Rev.2 

"Anti-Vibration Bar Assembly l/9",10/18/2006 

24. L5-04FU112 Rev.l 

"Anti-Vibration Bar Assembly 2/9", 8/22/2006 

25. L5-04FU113 Rev.O 

"Anti-Vibration Bar Assembly 3/9", 11/11/2005 

26. L5-04FU114 Rev.O 

"Anti-Vibration Bar Assembly 4/9", 11/11/2005 

27. L5-04FU115 Rev.2 

"Anti-Vibration Bar Assembly 5/9", 10/18/2006 

28. L5-04FU116 Rev.2 

"Anti-Vibration Bar Assembly 6/9", 10/18/2006 

29. L5-04FU117 Rev.9 

"Anti-Vibration Bar Assembly 7/9", 3/9/2007 

30. L5-04FU118 Rev.3 

"Anti-Vibration Bar Assembly 8/9", 4/6/2006 

31. L5-04FU119 Rev.l 

"Anti-Vibration Bar Assembly 9/9", 1/19/2006 

32. L5-04FU108 Rev.3 

"Tube Support Plate Assembly 3/3", 9/4/2006 

33. KAZ-04-0021 Rev.3 

"General Assembly l/3",8/10/2004 

34. KAZ-04-0022 Rev.3 

"General Assembly 2/3", 8/10/2004 

35. KAZ-04-0023 Rev.O 

"General Assembly 3/3", 2/15/2004 

36. L5-04GB001 Rev.9 

"Replacement Steam Generators Project Execution Plan", 8/4/2006 

37. SB-SO-FB-0003 Rev.7 

"Tubing and AVB Installation Procedure", 12/18/2007 

38. SB-SO-FB-0004 Rev.10 

"AVB Structure Assembly Procedure", 7/30/2007 

39. UGS-L5-050044 Rev.7 

"Anti-Vibration Bar Inspection Procedure (after assembling)", 5/21/2007 

40. UGS-L5-050045 Rev.10 

"Inspection Procedure for Tube and Anti-Vibration Bar Insertion", 11/2/2007 

41. UGNR-SON2-RSG-067 Rev.7 

"Nonconformance Report (Unacceptable gaps between Tubes and AVBs for 
Unit-2A)",9/2 1/2007 

42. UGNR-SON2-RSG-075 Rev.l 

"Nonconformance Report (Unacceptable gaps between Tubes and AVBs for 
Unit-2B)", 12/3/2007 



Root Cause Analysis Report for tube wear identified in the Unit 2 and Unit 3 Steam 
Generators of San Onofre Nuclear Generating Station 



[ , ) 

Attachment-7 UES-20120254 Rev.O (64/64) |Non-Proprietary| 



43. UGNR-SON3-RSG-024 Rev.l 

"Nonconformance Report for some Gaps between Tubes and AVBs are larger than 
the criterion", 4/24/2008 

44. UGNR-SON3-RSG-030 Rev.O 

"Nonconformance Report for some Gaps between Tubes and AVBs are larger than 
the criterion",3/20/2008 



Roof Cause Analysis Report for tube wear identified in the Unit 2 and Unit 3 Steam 
Generators of San Onofre Nuclear Generating Station 



